1. Field of the Invention
The invention relates to image processing methods and techniques, and more particularly to a method for defining coefficients of a convolution kernel for use in interpolating unknown pixel values.
2. Description of the Related Art
Digital cameras use an image sensor or a photosensitive device that is sensitive to a broad spectrum of light to capture an image of a scene. The photosensitive device reacts to light reflected from the scene and can translate the strength of that light into electronic signals that are digitized. Generally, an image sensor includes a two-dimensional array of light detecting elements, also called pixels, and generates electronic signals, also called pixel data, at each light detecting element that are indicative of the intensity of the light impinged upon each light detecting element. Thus, the sensor data generated by an image sensor is often represented as a two-dimensional array of pixel data.
For color applications, digital cameras are either implemented as multiple-image-sensor devices or single-image-sensor devices. In a multiple-image-sensor device, a digital camera or a video camera includes multiple image sensors (usually three), each sensor disposed to sense a different spectrum of the visible light (such as red, green and blue). Multiple-image-sensor devices are more complex and require precise optical design to ensure correct image registration. Therefore, the single-image-sensor implementation is more prevalent in digital cameras today.
To capture a color image in a single-image-sensor digital camera, a pattern of selectively transmissive filters is laid on top of the image sensor so that different groups of pixels are made to sense different color ranges. For example, a pattern of three types of filters may be used to sense three color ranges, such as the red, green and blue range of the visible spectrum. Alternately, the filters can be arranged in a “Bayer pattern” including individual luminance and chrominance sensing elements. For example, a four-color Bayer pattern including green1 and green2 luminance sensing elements, and red and blue chrominance sensing elements can be used. U.S. Pat. No. 3,971,065 to Bayer describes a color image sensor using an array of luminance and chrominance sensitive elements. The arrangement of color sensing elements in the color imaging array of the '065 patent is now commonly referred to as the “Bayer pattern.”
In a single-image-sensor digital camera, each pixel location is disposed to sample only one color component of the scene. Therefore, at each pixel location, pixel values for the other color components are missing. However, a full color image is usually represented by three set of pixel values, one set of pixel values for each of the three primary colors (such as red, green and blue) at every pixel location. Therefore, to create a full color picture from a single-image-sensor digital camera, an interpolation process is performed amongst the pixels to determine interpolated pixel values at each pixel location for the missing color components.
FIG. 1 illustrates the interpolation process for creating a full color image from sensor data captured using an image sensor overlaid with a pattern of color filters. Referring to FIG. 1, a pixel array 10 represents a portion of an image sensor having M rows and N columns of pixels. Specifically, pixel array 10 depicts a 5×5 neighborhood of pixels in the image sensor. A Bayer pattern of color filters is superimposed on pixel array 10 for capturing different color components of a scene. In the present illustration, a four-color Bayer pattern is used including a red filter superimposed on a pixel 12, a green1 filter superimposed on a pixel 14, a green2 filter superimposed on a pixel 16, and a blue filter superimposed on a pixel 18. This filter pattern repeats throughout the entire image sensor so that each pixel generates pixel data indicative of the impinging light intensity filtered by the respective color filter.
In FIG. 1, a full color image is represented in the RGB color space including a red color plane 20, a green color plane 22 and a blue color plane 24. A color pixel in the full color image can be expressed as a vector of pixel values consisting of pixel values for the red, green and blue color planes. For instance, a color pixel C(i, j) in a full color image may be expressed as follows:C(i, j)=[R(i, j)G(i, j)B(i, j)]T,where (i, j) are coordinates of a pixel in the image sensor, and R, G and B represent the pixel values for the respective color pixel in each of the three color planes. The interpolation process operates on the pixel data in pixel array 10 which includes pixel data of different colors sampled at different pixel locations and determines the missing pixel values so that each pixel location in pixel array 10 is represented by pixel values in each of three color planes 20, 22, 24 in the full color image.
Techniques for performing interpolation, sometimes referred to as “demosaicing,” are known in the art. For example, U.S. Pat. No. 4,642,678 to Cok, U.S. Pat. No. 5,373,322 to Laroche et al., and U.S. Pat. No. 5,475,769 to Wober et al. describe various methods for recovering missing pixel values from sampled color image data. The Wober patent describes a common approach using a convolution kernel to compute missing pixel values for each pixel location based on the pixel values of a neighborhood of pixels surrounding each pixel location. For an n×n neighborhood of pixels, the convolution kernel is an n×n set of coefficients.
FIG. 2 illustrates the interpolation process on a 5×5 neighborhood of pixels using a 5×5 convolution kernel. Referring to FIG. 2, pixel array 30 is a 5×5 neighborhood of pixels surrounding a center pixel 32. Pixel array 30 is overlaid with a Bayer filter pattern including four types of color filters (that is, red, blue, green1 and green2) in the same manner as pixel array 10 in FIG. 1. In the present illustration, center pixel 32 has been superimposed with a red filter. Therefore, the pixel value captured by center pixel 32 (shown as a value of “1”) represents the red color component of the light impinged upon that pixel location. Because the color of a scene generally does not change abruptly between adjacent pixel locations and also because the optics of the imaging device will always blur the impinging light so that a beam of light is spread out to adjacent pixels, the neighborhood of pixels surrounding center pixel 32 can be used to recover pixel values of the missing color components at center pixel 32.
In the interpolation process, a different kernel of coefficients is used for each color plane and for each filter pattern. In the present illustration, the missing pixel values to be reconstructed from the neighborhood of pixels are pixel values for the green and the blue color components. FIG. 2 illustrates a convolution kernel 34 for determining the missing pixel value for a given color plane, such as the green color component, of center pixel 32. An interpolated pixel value 36 is determined by multiplying each pixel value in the 5×5 neighborhood of pixels by its corresponding coefficient in convolution kernel 34, such as by performing a matrix multiplication. The products from the matrix multiplication operation are summed to arrive at the interpolated pixel value 36 for the green color component of center pixel 32 which in this case is 2.81.
The interpolation process requires a set of coefficients to be determined for each color plane in the full color image and for each type of filter patterns used by the image sensor. The Wober patent describes a method for determining a set of weighting coefficients using a linear minimum mean square error solution of a matrix and vector expression. The matrix and vector expression defines the relationship between a neighborhood of pixel values and the actual values of the same neighborhood of pixels for a particular color component. Other methods typically involve defining the coefficients manually based on the optical properties of light and the filter characteristics. The conventional methods for generating the coefficients of a convolution kernel are either computationally intensive or are imprecise. In general, the conventional methods do not provide acceptable interpolated pixel values and the resultant image often includes color distortion or fringes. Thus, an improved method for generating coefficients for a convolution kernel is desired.